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Amendments to the Claims; 

1 . (Currently amended) A computer-implemented m ethod of automatically 

controlling execution of an application program by a computer system, the application 
program having a graphical user interface for displaying information on a display coupled 
to the computer system comprising : 

capturing user input data and images displayed by the graphical user interface on the 
display during a recording phase of execution of the application progra m by the computer 
system ; 



analyzing the captured user input data and displayed images to generate an execution 
scenario script during the recording phase, wherein the captured input data is to be stored in a 
memory; 

generating simulated user input data based on the execution scenario script during a 
playback phase of execution of the application program by the computer system and inputting 
the simulated user input data to the application program; 

performing image analysis on images displayed by the graphical user interface on the 
display as a result of processing the simulated user input data during the playback phase and 
captured displayed images from the recording phase; 

checking recorded time conditions in the execution scenario script to handle variations in 
playback speed, and 



automatically controlling execution of the application program by the computer system 
based at least in part on the image analysis. 

2. (Original) The method of claim 1, wherein performing image analysis comprises 
comparing the displayed images captured during the recording phase with displayed images from 
the playback phase. 

3. (Original) The method of claim 1, wherein the user input data comprises at least one 
of keyboard input data and mouse input data. 
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4. (Original) The method of claim 1, wherein analyzing the captured user input data and 
displayed images in the recording phase comprises identifying actions of the application program 
corresponding to the captured user input data. 

5. (Currently amended) The method of claim 1, wherein the execution scenario script 
comprises a script including extended markup language (XML) tags. 

6. (Withdrawn) The method of claim 1, wherein analyzing the captured user input data 
and displayed images during a recording phase comprises: 

determining contours of objects shown in a displayed image; 
detecting an object of activity from among the objects; and 
detecting additional objects located adjacent to the object of activity. 

7. (Withdrawn) The method of claim 1, wherein performing image analysis comprises 
determining contours of objects in an image displayed by the graphical user interface during the 
playback phase and filtering the objects according to size to produce a set of hypotheses for 
active objects. 

8. (Withdrawn) The method of claim 7, wherein performing image analysis further 
comprises filtering objects according to space layout to produce a first subset of hypotheses for 
active objects. 

9. (Withdrawn) The method of claim 7, wherein performing image analysis further 
comprises filtering objects by content to produce a second subset of hypotheses for active 
objects. 

10. (Withdrawn) The method of claim 7, wherein performing image analysis further 
comprises structural filtering of objects to produce a best hypothesis for an active object. 
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1 1 . (Withdrawn) The method of claim 7, wherein performing image analysis further 
comprises recalculating old actions for a new object identified as the active object. 

12. (Currently amended) An article comprising: a machine computer r eadable medium 
containing instructions, which when executed, result in automatically controlling execution of an 
application program by a computer system, the application program having a graphical user 
interface for displaving information on a displav coupled to the computer svstem bv 

capturing user input data and images displayed by the graphical user interface interface 
on the display during a recording phase of execution of the application program by the computer 
system; 



analyzing the captured user input data and displayed images to generate an execution 
scenario script during the recording phase; 

generating simulated user input data based on the execution scenario script during a 
playback phase of execution of the application program by the computer system and inputting 
the simulated user input data to the application program; 

performing image analysis on images displayed by the graphical user interface on the 
display as a result of processing the simulated user input data during the playback phase and 
captured displayed images from the recording phase; 

checking recorded time conditions in the execution scenario script to handle variations in 
playback speed, and 



automatically controlling execution of the application program by the computer system 
based at least in part on the image analysis. 

13. (Original) The article of claim 12, wherein instructions for performing image 
analysis comprise instructions for comparing the displayed images captured during the recording 
phase with displayed images from the playback phase. 

14. (Original) The article of claim 12, wherein the user input data comprises at least one 
of keyboard input data and mouse input data. 
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15. (Original) The article of claim 12, wherein instructions for analyzing the captured 
user input data and displayed images in the recording phase comprise instructions for identifying 
actions of the application program corresponding to the captured user input data. 

16. (Currently amended) The article of claim 12, wherein the execution scenario script 
comprises a script including extended markup language (XML) tags. 

17. (Withdrawn) The article of claim 12, wherein instructions for analyzing the captured 
user input data and displayed images during a recording phase comprise instructions for: 

determining contours of objects shown in a displayed image; 
detecting an object of activity from among the objects; and 
detecting additional objects located adjacent to the object of activity. 

18. (Withdrawn) The article of claim 12, wherein instructions for performing image 
analysis comprise instructions for determining contours of objects in an image displayed by the 
graphical user interface during the playback phase and filtering the objects according to size to 
produce a set of hypotheses for active objects. 

19. (Withdrawn) The article of claim 18, wherein instructions for performing image 
analysis fiirther comprise instructions for filtering objects according to space layout to produce a 
first subset of hypotheses for active objects. 

20. (Withdrawn) The method of claim 18, wherein instructions for performing image 
analysis further comprise instructions for filtering objects by content to produce a second subset 
of hypotheses for active objects. 

21. (Withdrawn) The article of claim 18, wherein instructions for performing image 
analysis further comprise instructions for structural filtering of objects to produce a best 
hypothesis for an active object. 
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22. (Withdrawn) The article of claim 18, wherein instructions for performing image 
analysis further comprise instructions for recalculating old actions for a new object identified as 
the active object. 

23. (Currently presented) A cognitive control framework system for automatically 
controlling execution of an application program having a graphical user interface in a computer 
svstem comprising : 

a recording component adapted to capture user input data and images displayed by the 
graphical user interface during a recording phase of execution of the application program, and to 
analyze the captured user input data and displayed images to generate an execution scenario 
script during the recording phase, 

a playback component adapted to generate simulated user input data based on the 
execution scenario script during a playback phase of execution of the apphcation program, to 
input the simulated user input data to the application program, to perform image analysis on 
images displayed by the graphical user interface as a result of processing the simulated user input 
data during the playback phase and captured displayed images fi-om the recording phase; 
checking recorded time conditions in the execution scenario script to handle variations in 
playback speed, and to automatically control execution of the application program by the 
computer system b ased at least in part on the image analysis. 

24. (Original) The system of claim 23, wherein the playback component is adapted to 
compare the displayed images captured during the recording phase with displayed images fi-om 
the playback phase. 

25. (Original) The system of claim 23, w herein the recording component is adapted to 
identify actions of the application program corresponding to the captured user input data. 

26. (Withdrawn) The system of claim 23, wherein the recording component is adapted to 
determine contours of objects shown in a displayed image, detect an object of activity from 
among the objects, and detect additional objects located adjacent to the object of activity. 
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27. (Withdrawn) The system of claim 23, wherein the playback component is adapted to 
determine contours of objects in an image displayed by the graphical user interface during the 
playback phase and filter the objects according to size to produce a set of hypotheses for active 
objects. 

28. (Withdrawn) The system of claim 27, wherein the playback component is adapted to 
filter objects according to space layout to produce a first subset of hypotheses for active objects. 

29. (Withdrawn) The system of claim 27, wherein the playback component is adapted to 
filter objects by content to produce a second subset of hypotheses for active objects. 

30. (Withdrawn) The system of claim 27, wherein the playback component is adapted to 
structurally filter objects to produce a best hypothesis for an active object. 

31. -33. (Cancelled) 
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